Moving image reproduction apparatus having function of correcting camera shake during moving image reproduction, method of controlling the same, and storage medium

ABSTRACT

A moving image reproduction apparatus that reproduces and displays captured moving image on a display device. An MPU acquires a shake amount, and determines whether the shake amount exceeds a range within which camera shake in the captured moving image can be corrected. When the shake amount exceeds the range, the MPU calculates an interpolated shake amount based on the shake amount such that the interpolated shake amount is within the range. The MPU decides a camera shake correction amount based on the shake amount and the interpolated shake amount. The MPU corrects camera shake using the camera shake correction amount. When the shake amount does not exceed the range, the MPU decides the shake amount as the camera shake correction amount, whereas when the shake amount exceeds the range, the MPU decides the interpolated shake amount as the camera shake correction amount.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a moving image reproduction apparatus,a method of controlling the same, and a storage medium.

Description of the Related Art

Due to recent demand for size reduction, a digital video camera, adigital still camera, and so forth are reduced in space for disposing asensor circuit board on which an image pickup device is mounted, and amain circuit board on which electronic components for processing controlsignals and image signals are mounted, and are also reduced in distancebetween the circuit boards. Further, due to advances in board packagingtechnology, packaging density of electronic components mounted on thecircuit boards has also increased.

When video is shot by an image pickup apparatus, such as a digital videocamera, camera shake is sometimes included in the stored video due toshake (vibration), such as hand shake, of the image pickup apparatus. Toovercome this problem, there have been proposed various techniques forcorrecting camera shake in video when the stored video is reproduced.For example, a technique is proposed in which a motion vector indicativeof camera shake in video is detected, and a partial video area isselected according to the detected motion vector, for enlarged displayof the selected partial video area (see Japanese Laid-Open PatentPublication (Kokai) No. H07-143380). Further, a technique is proposed inwhich shake of an image pickup apparatus is detected by a sensor duringshooting, and data of the detected shake is stored together with videodata so as to correct camera shake in the video using the shake dataduring reproduction of the video (see Japanese Laid-Open PatentPublication (Kokai) No. H10-42233). Furthermore, a technique is proposedin which the display magnification of enlarged display for camera shakecorrection is determined, based on the magnitude of a shake amount ofthe shake data, at the time of video reproduction (see JapaneseLaid-Open Patent Publication (Kokai) No. 2012-44418).

In the above-described conventional techniques, however, a sudden(momentary) large camera shake caused by instantaneous impact applieddirectly or indirectly to the image pickup apparatus during videoshooting is also stored as shake data together with video data. If thevideo data is subjected to camera shake correction during reproductionof the same, camera shake correction processing is performed also on thesudden large camera shake, and hence, the enlargement ratio of thereproduced video is increased in accordance with execution of the camerashake correction processing, which results in degradation of imagequality. Further, in a case where the shake amount is large, an angle ofview of the reproduced video largely changes from the angle of view atthe time of shooting the video.

To solve this problem, if an upper limit is set for the enlargementratio of the reproduced video so as to prevent a large change in theangle of view and the camera shake correction is only allowed tofunction in the case of occurrence of camera shake which is not so largeas will make the enlargement ratio of the reproduced video for thecamera shake correction larger than the upper limit, the shake blur isnot conspicuous to a viewer of the reproduced video. However, if camerashake occurs which makes the enlargement ratio of the reproduced imagelarger than the upper limit, the camera shake cannot be corrected, andhence a sudden shake blur occurs in the reproduced video, causing theviewer to feel a strangeness.

SUMMARY OF THE INVENTION

The present invention provides a moving image reproduction apparatusthat is capable of reproducing captured moving image which makes itdifficult for a viewer to feel a strangeness, even when a sudden camerashake occurred during moving image shooting, by performing camera shakecorrection during captured moving image reproduction such that the angleof view is not made largely different from that at the time of movingimage shooting.

In a first aspect of the present invention, there is provided a movingimage reproduction apparatus that reproduces and displays capturedmoving image on a display device, comprising at least one processor thatexecutes a program stored in a memory to function as an acquisition unitconfigured to acquire a shake amount associated with the captured movingimage, a determination unit configured to determine whether or not theshake amount exceeds a range within which camera shake in the capturedmoving image can be corrected, a calculation unit configured to, in acase where the shake amount exceeds the range, calculate an interpolatedshake amount based on the shake amount such that the interpolated shakeamount is within the range, a decision unit configured to decide acamera shake correction amount for correcting the camera shake in thecaptured moving image based on the shake amount and the interpolatedshake amount, and a correction unit configured to correct the camerashake in the captured moving image using the camera shake correctionamount, wherein in a case where the shake amount does not exceed therange, the decision unit decides the shake amount as the camera shakecorrection amount, whereas in the case where the shake amount exceedsthe range, the decision unit decides the interpolated shake amount asthe camera shake correction amount.

In a second aspect of the present invention, there is provided a methodof controlling a moving image reproduction apparatus that reproduces anddisplays captured moving image on a display device, comprising acquiringa shake amount associated with the captured moving image, determiningwhether or not the shake amount exceeds a range within which camerashake in the captured moving image can be corrected, calculating, in acase where the shake amount exceeds the range, an interpolated shakeamount based on the shake amount such that the interpolated shake amountis within the captured moving image, deciding a camera shake correctionamount for correcting the camera shake in the captured moving imagebased on the shake amount and the interpolated shake amount, andcorrecting the camera shake in the captured moving image using thecamera shake correction amount, wherein said deciding includes deciding,in a case where the shake amount does not exceed the range, the shakeamount as the camera shake correction amount, and deciding, in the casewhere the shake amount exceeds the range, the interpolated shake amountas the camera shake correction amount.

In a third aspect of the present invention, there is provided anon-transitory computer-readable storage medium storing acomputer-executable program for executing a method of controlling amoving image reproduction apparatus that reproduces and displayscaptured moving image on a display device, wherein the method comprisesacquiring a shake amount associated with the captured moving image,determining whether or not the shake amount exceeds a range within whichcamera shake in the captured moving image can be corrected, calculating,in a case where the shake amount exceeds the range, an interpolatedshake amount based on the shake amount such that the interpolated shakeamount is within the range, deciding a camera shake correction amountfor correcting the camera shake in the captured moving image based onthe shake amount and the interpolated shake amount, and correcting thecamera shake in the captured moving image using the camera shakecorrection amount, wherein said deciding includes deciding, in a casewhere the shake amount does not exceed the range, the shake amount asthe camera shake correction amount, and deciding, in the case where theshake amount exceeds the range, the interpolated shake amount as thecamera shake correction amount.

According to the present invention, it is possible to reproduce capturedmoving image which makes it difficult for a viewer to feel astrangeness, even when a sudden camera shake occurred during movingimage shooting, by performing camera shake correction during capturedmoving image reproduction such that the angle of view is not madelargely different from that at the time of moving image shooting.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a control block diagram of component elements of a digitalvideo camera as a video reproduction apparatus according to anembodiment of the present invention, which perform processing fromshooting to storing of video.

FIG. 2 is a control block diagram of component elements of the digitalvideo camera, which perform reproduction processing of stored video.

FIG. 3 is a flowchart of a video data reproduction process performed bythe digital video camera.

FIG. 4A is a diagram showing changes with time in a reference shakeamount and an interpolated shake amount, on a frame basis.

FIG. 4B is a partial enlarged view of FIG. 4A.

FIG. 5A is a diagram showing a table indicating a relationship betweenthe reference shake amount and the interpolated shake amount.

FIG. 5B is a graph formed from the table shown in FIG. 5A.

FIG. 6 is a flowchart of a camera shake correction amount-decidingprocess executed in a step in the video data reproduction process inFIG. 3.

DESCRIPTION OF THE EMBODIMENTS

The present invention will now be described in detail below withreference to the accompanying drawings showing embodiments thereof.

FIG. 1 is a control block diagram of component elements of a digitalvideo camera (hereinafter referred to as the “video camera”) 100 as avideo reproduction apparatus (moving image reproduction apparatus)according to an embodiment of the present invention, which performprocessing from shooting to storing of video (moving image). The videocamera 100 includes a lens unit 101, an image pickup device 102, ananalog signal processor 103, a camera signal processor 104, and anencoder 105. Further, the video camera 100 includes an angular velocitysensor 111, a shake data calculation section 112, a clock 113, a timecode-setting section 114, a metadata setting section 115, a storagecontroller 121, and a storage medium 122.

The lens unit 101 causes an object image to be formed on the imagepickup device 102. The image pickup device 102, which is implementede.g. by a CCD sensor or a CMOS sensor, photoelectrically converts anobject image formed thereon to output analog signals. The analog signalprocessor 103 performs predetermined processing on the analog signalsoutput from the image pickup device 102 to generate analog imagesignals. The analog signal processor 103 includes e.g. a CDS (correlateddouble sampling) circuit, an AGC (automatic gain control) circuit, andso forth. The camera signal processor 104 includes an analog-to-digitalconverter, and generates digital video signals from the analog imagesignals generated by the analog signal processor 103. The encoder 105encodes the digital video signals generated by the camera signalprocessor 104 in MPEG2 (Moving Picture Experts Group 2) format to outputa stream in the MPEG2 format.

The angular velocity sensor 111 detects the angular velocity of shakeapplied to the video camera 100. The shake data calculation section 112calculates a shake amount of the video camera 100 based on an outputfrom the angular velocity sensor 111 to output shake data including thecalculated shake amount to the metadata setting section 115. The timecode-setting section 114 sets a time code at the time of storing video,based on time information acquired from the clock 113. The metadatasetting section 115 outputs the shake data, the time code,camera-related data (not shown), and so forth, as metadata. The storagecontroller 121 stores the stream (video data) in the MPEG2 format fromthe encoder 105 in a state in which the metadata from the metadatasetting section 115 is added thereto, in the storage medium 122.

FIG. 2 is a control block diagram of component elements of the videocamera 100, which perform reproduction processing of video (capturedmoving image) stored in the storage medium 122. The storage controller121 and the storage medium 122 are component elements involved in theprocessing from shooting to storing of video, and also involved in thereproduction processing of the stored video. The video camera 100further includes a microcomputer 210 (hereinafter referred to as the“MPU 210”), a decoder 221, a memory 222, an image processor 223, and adisplay device 224.

As mentioned above, the storage medium 122 stores the video data havingthe metadata including the shake data added thereto. Note that theformat for storing metadata and video data is not limited, but anysuitable format may be used insofar as it is capable of storing shakedata and video data acquired by the video camera 100. For example,metadata may be stored separately from video data. The storagecontroller 121 acquire the video data and the metadata (shake data) fromthe storage medium 122. The decoder 221 decodes the video data (streamin the MPEG2 format) acquired by the storage controller 121 into digitalvideo signals to output the digital video signals to the memory 222together with the metadata.

The memory 222 stores e.g. a several-second length of the decodeddigital video signals. By storing the several-second length of thedigital video signals in the memory 222, it is possible to obtaininformation preceding in time a reproduced image (unit image beingreproduced) which is being output to the display device 224. The imageprocessor 223 reproduces the video data according to a cutout positionand a cutout size determined by the MPU 210. More specifically, theimage processor 223 sequentially cuts out, from a plurality of unitimages (field images or frame images) which form the video representedby the video data, respective images each having a predetermined cutoutsize and cut out from a predetermined cutout position. Then, the imageprocessor 223 subjects each cut-out image to electronic zoom to outputthe resulting image to the display device 224. The display device 224displays the video by sequentially displaying the images acquired fromthe image processor 223.

The MPU 210 controls the overall operation of the video camera 100. Notethat although not shown in FIG. 1, the MPU 210 not only controls thereproduction processing of video but also controls the processing fromshooting to storing of video by the video camera 100. The MPU 210includes a motion vector detection section 201, a cutoutposition-setting section 202, a metadata readout section 203, a shakedata acquisition section 204, and a cutout size-setting section 205.Further, the MPU 210 includes a time code-setting section 206, a readoutaddress-setting section 207, a panning component removal section 208,and a camera shake correction amount-deciding section 209. Note thatFIG. 2 shows only essential components of the MPU 210 for performingcamera shake correction, and the functions of the MPU 210 are notlimited to the functions of the above-described sections.

The motion vector detection section 201 detects a motion vector fromdigital images stored in the memory 222. The motion vector detected bythe motion vector detection section 201 plays the role of an index ofcamera shake in video, and is output to the camera shake correctionamount-deciding section 209. The metadata readout section 203 reads outmetadata from a readout address of the memory 222 set by the readoutaddress-setting section 207. The shake data acquisition section 204acquires shake data from the metadata acquired by the metadata readoutsection 203 to output the shake data to the camera shake correctionamount-deciding section 209. The time code-setting section 206 acquiresa time code of a current reproduced image (unit image being reproduced)from the metadata readout section 203, and notifies the readoutaddress-setting section 207 of a time code indicative of a time apredetermined time period after the acquired time code.

The readout address-setting section 207 sets an address corresponding tothe time code notified from the time code-setting section 206 as areadout address of the memory 222. This makes it possible for themetadata readout section 203 to acquire metadata of an image (unitimage) to be reproduced at a time the predetermined time period afterthe reproduction of the current image.

The panning component removal section 208 performs filtering forremoving low-frequency components from the shake data acquired from theshake data acquisition section 204 (to remove a panning component fromthe shake data), and returns the shake data having the panning componentremoved therefrom to the shake data acquisition section 204. The cutoutsize-setting section 205 determines a cutout size of the unit imagebased on the shake data acquired by the shake data acquisition section204, and notifies the cutout size to the image processor 223. The camerashake correction amount-deciding section 209 determines a camera shakecorrection amount based on the output from the motion vector detectionsection 201 and the output from the shake data acquisition section 204.The cutout position-setting section 202 determines (moves), based on thecamera shake correction amount decided by the camera shake correctionamount-deciding section 209, the cutout position of the unit image suchthat camera shake in video is corrected, and notifies the resultingcutout position to the image processor 223.

FIG. 3 is a flowchart of a video data reproduction process performed bythe video camera 100. The video data reproduction process shown in FIG.3 is realized by the MPU 210 executing a predetermined program forreproducing and displaying video data and the sections of the MPU 210cooperating with each other to control the operations of the sections ofthe video camera 100.

Referring to FIG. 3, in a step S301, before the storage controller 121reads out first frame data of reproduction target video from the storagemedium 122, the shake data acquisition section 204 acquires a correctionamount upper limit for correcting camera shake in video via the metadatareadout section 203. The correction amount upper limit is an upper limitvalue of a correction amount range within which camera shake in videocan be corrected. Here, it is assumed that a predetermined value is setin advance as the upper limit value. In a step S302, the shake dataacquisition section 204 acquires shake data in an amount correspondingto a predetermined time period (a predetermined number of frames) from astart point of reproduction of the video data, via the metadata readoutsection 203.

In a step S303, the panning component removal section 208 acquires theshake data from the shake data acquisition section 204, and performsfiltering for removing low-frequency components (panning component) fromthe acquired shake data. Thus, movement of an object caused by thepanning of the video camera 100 is removed from the camera shake to becorrected. In a step S304, the motion vector detection section 201divides the video data into a grid (lattice), and detects a motionvector from data indicative of inter-frame correlations, for eachrectangle area of the grid. In a step S305, the camera shake correctionamount-deciding section 209 performs a camera shake correctionamount-deciding process, described hereinafter with reference to FIG. 6,to decide a camera shake correction amount for use in videoreproduction, based on the shake data having the panning componentremoved therefrom and the motion vector detected in the step S304.

In a step S306, the cutout size-setting section 205 checks the amplitudeof the camera shake represented by the shake data having the panningcomponent removed therefrom in the step S303, and determines a cutoutsize of an image in each frame such that a change in the cutout size iswithin a predetermined range. Further, the cutout position-settingsection 202 determines (moves) the cutout position such that camerashake in video is corrected, based on the camera shake correction amountdetermined in the step S305, and sets the cutout position in the imageprocessor 223. The determination of the cutout position can be performedusing a desired known technique, and hence detailed description thereofis omitted. When the cutout size and the cutout position are set in thestep S306, the image processor 223 cuts out an image having the cutoutsize set in the step S306 from the set cutout position of the unitimage, and outputs the cut-out image to the display device 224.

In a step S307, the MPU 210 determines whether or not the reproductionof the video has been terminated (the video has been reproduced to thelast frame). If it is determined that the reproduction of the video hasnot been terminated (NO to the step S307), the MPU 210 returns to thestep S302, whereas if it is determined that the reproduction of thevideo has been terminated (YES to the step S307), the MPU 210 terminatesthe present process.

Next, a description will be given of a method of deciding the camerashake correction amount by the camera shake correction amount-decidingprocess in the step S305. To decide the camera shake correction amount,the correction amount upper limit as the upper limit of the correctionamount range within which camera shake in video can be corrected and anactual shake amount associated with the frame number of a framecurrently being reproduced (hereinafter referred to as the “referenceshake amount”) are used. In a case where the reference shake amountexceeds the correction amount upper limit, interpolation processing isperformed to determine an interpolated shake amount, and the determinedinterpolated shake amount is decided as the camera shake correctionamount. On the other hand, in a case where the reference shake amount isnot larger than the correction amount upper limit, the reference shakeamount is directly decided as the camera shake correction amount.

FIG. 4A is a diagram showing an example of changes with time in thereference shake amount Bcur(n) and the interpolated shake amountBnxt(n), on a frame basis. FIG. 4B is a partial enlarged view of FIG.4A. FIG. 5A is a diagram showing a table indicating a relationshipbetween the reference shake amount Bcur(n) and the interpolated shakeamount Bnxt(n). FIG. 5B is a graph formed from the table shown in FIG.5A. In FIG. 5B, the reference shake amount Bcur(n) is indicated by abroken line, and the interpolated shake amount Bnxt(n) is indicated by asolid line.

The correction amount upper limit Bmax is set in advance to apredetermined value, and in the present embodiment, it is assumed to beset to “100”. In a range not larger than the correction amount upperlimit Bmax, the reference shake amount Bcur(n) is decided as the camerashake correction amount.

On the other hand, in a case where the reference shake amount Bcur(n) islarger than the correction amount upper limit Bmax, the interpolatedshake amount Bnxt(n) is determined from reference shake amountsBcur(n±3), and the determined interpolated shake amount Bnxt(n) is usedas the camera shake correction amount. Although detailed descriptionwill be given hereinafter, in the present embodiment, a current framenumber is represented by n, and the interpolated shake amount Bnxt(n) isdetermined by performing interpolation processing using the referenceshake amounts Bcur(n±3) of each three frames preceding and following theframe having the current frame number n. Note that the reference shakeamounts Bcur(n±3) represent six reference shake amounts Bcur(n−3),Bcur(n−2), Bcur(n−1), Bcur(n+1), Bcur(n+2), and Bcur(n+3).

Note that to calculate the interpolated shake amount Bnxt(n), it is notnecessarily required to use the reference shake amounts Bcur(n±3) ofeach three preceding and following frames, but it is possible todetermine the number of preceding and following frames by taking intoaccount interpolation accuracy and computation load. For example, byacquiring at least three reference shake amounts, i.e. a reference shakeamount of an n-th frame, and reference shake amounts of n−1-th andn+1-th frames respectively preceding and following the n-th frame, it ispossible to calculate the reference shake amount of the n-th frame byinterpolation. Further, the number of acquired reference shake amountsof preceding frames and the number of acquired reference shake amountsof following frames with reference to the frame having the frame numbern may be different. In short, by performing interpolation processingusing reference shake amounts Bcur(n±1) of each at least one framepreceding and following the frame having the frame number n, it ispossible to determine the interpolated shake amount Bnxt(n).

FIG. 6 is a flowchart of the camera shake correction amount-decidingprocess executed in the step S305. Next, the camera shake correctionamount-deciding process will be described, by referring to FIGS. 4A to5B, as required. In a step S601, the camera shake correctionamount-deciding section 209 determines the correction amount upper limitBmax. The correction amount upper limit Bmax can be determined by aknown technique, and hence description of a method of determining thesame is omitted. In a step S602, the camera shake correctionamount-deciding section 209 acquires the reference shake amount Bcur(n)corresponding to the current frame number n, from the shake data and themotion vector. In a step S603, the camera shake correctionamount-deciding section 209 acquires the reference shake amountsBcur(n±3) of the each three frames preceding and following the framehaving the current frame number n. In a step S604, the camera shakecorrection amount-deciding section 209 determines whether or not thereference shake amount Bcur(n) is larger than the correction amountupper limit Bmax.

If it is determined that the reference shake amount Bcur(n) is notlarger than the correction amount upper limit Bmax (NO to the stepS604), the camera shake correction amount-deciding section 209 proceedsto a step S612. In the step S612, the camera shake correctionamount-deciding section 209 sets the reference shake amount Bcur(n) asthe camera shake correction amount of the frame having the current framenumber n. After the camera shake correction amount is set in the stepS612, the camera shake correction amount-deciding section 209 terminatesthe present process, followed by executing the step S306 of the videodata reproduction process in FIG. 3.

If it is determined in the step S604 that the reference shake amountBcur(n) is larger than the correction amount upper limit Bmax (YES tothe step S604), the camera shake correction amount-deciding section 209proceeds to a step S605. In the step S605, the camera shake correctionamount-deciding section 209 acquires a camera shake correction amount ofthe preceding frame having the frame number (n−1). The camera shakecorrection amount acquired in the step S605 is either the Bcur(n−1) orthe Bnxt (n−1). In a step S606, the camera shake correctionamount-deciding section 209 acquires the maximum value of the referenceshake amounts Bcur(n±3) (hereinafter referred to as the “neighboringmaximum value Bpeak”). In the example illustrated in FIGS. 4A and 4B,the reference shake amount Bcur(n+1) is acquired as the neighboringmaximum value Bpeak.

In a step S607, the camera shake correction amount-deciding section 209determines whether or not at least one value of the reference shakeamounts Bcur(n±3) is not larger than the correction amount upper limitBmax. If it is determined that at least one value of the reference shakeamounts Bcur(n±3) is not larger than the correction amount upper limitBmax (YES to the step S607), the camera shake correction amount-decidingsection 209 proceeds to a step S608. In the step S608, the camera shakecorrection amount-deciding section 209 sets one of the reference shakeamounts Bcur(n±3), which is not larger than the correction amount upperlimit Bmax and also is the maximum value of the reference shake amounts,as the latest shake amount Bprev. On the other hand, if it is determinedthat all of the reference shake amounts Bcur(n±3) are larger than thecorrection amount upper limit Bmax (NO to the step S607), the camerashake correction amount-deciding section 209 proceeds to a step S609. Inthe step S609, the camera shake correction amount-deciding section 209sets the camera shake correction amount of the preceding frame havingthe frame number (n−1) as the latest shake amount Bprev. In a step S610,the camera shake correction amount-deciding section 209 determines theinterpolated shake amount Bnxt(n) from the reference shake amountBcur(n), the correction amount upper limit Bmax, the neighboring maximumvalue Bpeak, and the latest shake amount Bprev, using a linear function.

Here, a description will be given of a method of determining theinterpolated shake amount Bnxt(n) with reference to FIG. 4B. Referringto FIG. 4B, the reference shake amounts Bcur(n) and Bcur(n+1) are bothlarger than the correction amount upper limit Bmax, and hence theinterpolated shake amounts Bnxt(n) and Bnxt(n+1) are calculated.Further, the reference shake amount Bcur(n+1) is the neighboring maximumvalue Bpeak. Differences D1 and D2 between the correction amount upperlimit Bmax, and the reference shake amounts Bcur(n) and Bcur(n+1) of theframes having the respective frame numbers n and n+1 are determined byrespective equations (1) and (2), described below. Differences D3 and D4between the latest shake amount Bprev, and the interpolated shakeamounts Bnxt(n) and Bnxt(n+1) of the frames having the respective framenumbers n and n+1 are determined by respective equations (3) and (4),described below. Assuming that the difference D1 and the difference D2,and the difference D3 and the difference D4 are in proportionalrelationship with each other, respectively, an equation (5), describedbelow, holds, and the interpolated shake amount Bnxt(n) can bedetermined by an equation (6), described below, which is changed fromthe equation (5).D1=Bcur(n)−Bmax  (1)D2=Bcur(n+1)−Bmax=Bpeak−Bmax  (2)D3=Bnxt(n)−Bcur(n−1)=Bnxt(n)−Bprev  (3)D4=Bnxt(n+1)−Bcur(n−1)=Bmax−Bprev  (4)(Bcur(n)−Bmax):(Bpeak−Bmax)=(Bnxt(n)−Bprev):(Bmax−Bprev)  (5)Bnxt(n)={[(Bcur(n)−Bmax)×(Bmax−Bprev)]/(Bpeak−Bmax)}+Pprev  (6)

In a step S611, the camera shake correction amount-deciding section 209sets the interpolated shake amount Bnxt(n) calculated in the step S611as the camera shake correction amount of the frame having the framenumber n. When the camera shake correction amount is set in the stepS611, the camera shake correction amount-deciding section 209 terminatesthe present process, followed by executing the step S306 of the videodata reproduction process in FIG. 3. That is, the camera shakecorrection amount is decided in one of the steps S611 and S612, and thestep S306 is executed using the decided camera shake correction amount.Note that although the description has been given of the case where theshake amounts assume positive values, with reference to FIGS. 4A to 6,there is a case where the shake amounts assume negative values. In sucha case, positive and negative signs are only reversed from the casewhere the shake amounts assume positive values, so that the camera shakecorrection amount can be determined by the same method as employed inthe case of the shake amounts assuming positive values.

According to the above-described embodiment, in the camera shakecorrection process performed during reproduction of video, even in acase where the video includes a scene in which a large camera shakeexceeding the correction amount upper limit (correctable range) hasoccurred, the camera shake correction process is performed withoutexcessive enlargement of the video. In this case, a camera shakecorrection amount of a frame of which the actual shake amount (referenceshake amount) exceeds the correction amount upper limit is determined byinterpolation processing performed by taking into account shake amountsof neighboring frames. This makes it possible to progressively reduceeffects of camera shake correction starting from the vicinity of thecorrection amount upper limit, instead of suddenly eliminating theeffects of camera shake correction after the correction amount upperlimit is reached. Therefore, it is possible to suppress occurrence of anunnatural camera shake in video during video reproduction, therebymaking it difficult for a user to feel a strangeness.

Note that the above description is given of the method of performingcamera shake correction by comparing the reference shake amount Bcur(n)with the correction amount upper limit Bmax, and deciding theinterpolated shake amount Bnxt(n) as the camera shake correction amountin a case where the reference shake amount Bcur(n) is larger than thecorrection amount upper limit Bmax. However, the method of determiningthe camera shake correction amount by taking the correction amount upperlimit Bmax into account is by no means limited to the method describedabove, but any other suitable method may be employed. Further, in theabove-described embodiment, the interpolated shake amount Bnxt(n) iscalculated from the reference shake amounts of each three framespreceding and following the current frame, by subjecting the referenceshake amount Bcur(n) to interpolation using a linear function. However,the method of subjecting the reference shake amount Bcur(n) that exceedsthe correction amount upper limit Bmax to interpolation is not limitedto this, but any other suitable method may be employed. Furthermore,although in the above-described embodiment, the correction amount upperlimit Bmax is determined before video reproduction, the correctionamount upper limit Bmax may be determined by any other suitable method.For example, the maximum shake amount and the average shake amounts ofthe whole video may be detected in advance and the correction amountupper limit Bmax may be changed according to results of the detection ona video-by-video basis. Alternatively, the correction amount upper limitBmax may be manually set before reproduction of the video.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

For example, although in the above-described embodiment, a video camerais described as the video reproduction apparatus, this is notlimitative, but the present invention can also be applied to anelectronic apparatus, such as a personal computer or a smartphone, whichhas a function of reproducing video. Further, although in theabove-described embodiment, the description has been given of a casewhere video is reproduced which is stored in the storage medium 122mounted on the video camera 100, video data of video to be reproducedmay be acquired via a network.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-Ray Disc (BD)™),a flash memory device, a memory card, and the like.

This application claims the benefit of Japanese Patent Application No.2017-080708 filed Apr. 14, 2017 which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. A moving image reproduction apparatus thatreproduces and displays captured moving image on a display device,comprising at least one processor that executes a program stored in amemory to function as: an acquisition unit configured to acquire a shakeamount associated with the captured moving image; a determination unitconfigured to determine whether or not the shake amount exceeds a rangewithin which camera shake in the captured moving image can be corrected;a calculation unit configured to, in a case where the shake amountexceeds the range, calculate an interpolated shake amount based on theshake amount such that the interpolated shake amount is within therange; a decision unit configured to decide a camera shake correctionamount for correcting the camera shake in the captured moving imagebased on the shake amount and the interpolated shake amount; and acorrection unit configured to correct the camera shake in the capturedmoving image using the camera shake correction amount, wherein in a casewhere the shake amount does not exceed the range, the decision unitdecides the shake amount as the camera shake correction amount, whereasin the case where the shake amount exceeds the range, the decision unitdecides the interpolated shake amount as the camera shake correctionamount.
 2. The moving image reproduction apparatus according to claim 1,wherein the moving image is formed by a plurality of frames, and whereinthe acquisition unit acquires the shake amount on a frame-by-framebasis.
 3. The moving image reproduction apparatus according to claim 2,wherein in a case where a shake amount of a predetermined frame of theplurality of frames exceeds the range, the calculation unit calculatesthe interpolated shake amount by calculating the shake amount of thepredetermined frame by interpolation using shake amounts of framespreceding and following the predetermined frame.
 4. The moving imagereproduction apparatus according to claim 2, wherein the correction unitincludes a setting unit configured to set a cutout position of an imageon a frame-by-frame basis such that the camera shake in the capturedmoving image is corrected based on the camera shake correction amountdecided by the decision unit.
 5. The moving image reproduction apparatusaccording to claim 1, further comprising a removal unit configured toremove low-frequency components from the shake amount acquired by theacquisition unit, and wherein the acquisition unit outputs the shakeamount having the low-frequency components removed therefrom by theremoval unit to the calculation unit.
 6. A method of controlling amoving image reproduction apparatus that reproduces and displayscaptured moving image on a display device, comprising: acquiring a shakeamount associated with the captured moving image; determining whether ornot the shake amount exceeds a range within which camera shake in thecaptured moving image can be corrected; calculating, in a case where theshake amount exceeds the range, an interpolated shake amount based onthe shake amount such that the interpolated shake amount is within therange; deciding a camera shake correction amount for correcting thecamera shake in the captured moving image based on the shake amount andthe interpolated shake amount; and correcting the camera shake in thecaptured moving image using the camera shake correction amount, whereinsaid deciding includes deciding, in a case where the shake amount doesnot exceed the range, the shake amount as the camera shake correctionamount, and deciding, in the case where the shake amount exceeds therange, the interpolated shake amount as the camera shake correctionamount.
 7. A non-transitory computer-readable storage medium storing acomputer-executable program for executing a method of controlling amoving image reproduction apparatus that reproduces and displayscaptured moving image on a display device, wherein the method comprises:acquiring a shake amount associated with the captured moving image;determining whether or not the shake amount exceeds a range within whichcamera shake in the captured moving image can be corrected; calculating,in a case where the shake amount exceeds the range, an interpolatedshake amount based on the shake amount such that the interpolated shakeamount is within the range; deciding a camera shake correction amountfor correcting the camera shake in the captured moving image based onthe shake amount and the interpolated shake amount; and correcting thecamera shake in the captured moving image using the camera shakecorrection amount, wherein said deciding includes deciding, in a casewhere the shake amount does not exceed the range, the shake amount asthe camera shake correction amount, and deciding, in the case where theshake amount exceeds the range, the interpolated shake amount as thecamera shake correction amount.